USE 557 - identify records for harvest and parse bitstream download information by ghukill · Pull Request #9 · MITLibraries/dspace-fulltext-harvester

ghukill · 2026-05-13T19:37:53Z

Purpose and background context

This PR introduces the mechanism by which this application will identify records to harvest fulltext for. And, a lightweight DSpace client to generate pre-signed URLs for known bitstream UUIDs.

For now, this application only wants to harvest fulltext bitstreams for DSpace theses. Ultimately, it will want to write the fulltext back to the TIMDEX dataset under a new data type called "fulltexts" (see USE-531), associated with specific TIMDEX records from the "records" data type. As such, it makes sense to read records from the TIMDEX dataset as the primary driver for what to harvest.

Additionally, the DSpace theses records in the TIMDEX dataset have their original DSpace METS which conveniently includes bitstream information! By utilizing that, we can avoid 3-4 DSpace APIs calls per item, and skip directly to generating a pre-signed S3 URL for a known bitstream UUID as the only DSpace API call we'll need to make.

Functionality added:

Identify DSpace records to harvest fulltext for
Extracting bitstream information from those record's METS files
Lighweight DSpace client that supports authentication + pre-signed URL generation

Functionality coming based on this:

A harvesting loop that performs all above
Downloading of the bitstream with a pre-signed URL in hand
Writing fulltext back to TIMDEX dataset

How can a reviewer manually see the effects of these changes?

1- Test new class `TIMDEXThesesRecords`

Set Dev1 TimdexManagers credentials.

Create .env file:

TDA_LOG_LEVEL=DEBUG
WARNING_ONLY_LOGGERS=asyncio,botocore,urllib3,s3transfer,boto3,MARKDOWN
TIMDEX_DATASET_LOCATION=s3://timdex-extract-dev-222053980223/dataset

Start ipython shell:

uv run --env-file .env ipython

Init TIMDEXThesesRecords instance, limited to 10 records:

import logging
from dfh.timdex_dataset import TIMDEXThesesRecords

# bootstrap logging
logging.basicConfig(level=logging.DEBUG)

tdt = TIMDEXThesesRecords(limit=10)

Note that no run_id filtering is applied here, thereby defaulting to all, current dspace records that are deemed to be theses.

Retrieve all records as a list from the iterator:

records = list(tdt.record_and_bitstream_metadata_iter())

Observe first record:

records[0]
"""
{'timdex_record_id': 'dspace:1721.1-150510',
 'run_id': '5e11f27e-fe05-44f4-8bcb-5acbdb032cf3',
 'run_record_offset': 100001,
 'run_date': datetime.datetime(2026, 5, 13, 0, 0),
 'run_timestamp': datetime.datetime(2026, 5, 13, 9, 49, 33, 846930, tzinfo=<DstTzInfo 'America/Detroit' EDT-1 day, 20:00:00 DST>),
 'fulltext_bitstream': {'uuid': 'b0e3c854-de3e-44ce-94a3-6088615a5eeb',
  'href': 'https://..../bitstreams/b0e3c854-de3e-44ce-94a3-6088615a5eeb/download',
  'size': 74970,
  'checksum': '2e3d96be8538d9c1a549d7292055ee78',
  'checksum_type': 'MD5',
  'mimetype': 'text/plain'}}
"""

Note: ignore the testing DSpace host found in the record.

With this in hand from this class + iterator, we have enough information to:

Request a pre-signed S3 URL for the bitstream UUID b0e3c854-de3e-44ce-94a3-6088615a5eeb
Download that fulltext
Write fulltext back to TIMDEX dataset associated with this specific record (timdex_record_id="dspace:1721.1-150510", run_id="5e11f27e-fe05-44f4-8bcb-5acbdb032cf3", run_record_offset="100001")

This iterator is quite quick. The I/O bottleneck will be in future work of parallelizing the requests for pre-signed bitstream URLs from this information.

Note

This section has been updated based on our pivot to using dspace-rest-python client versus a lightweight, custom client for DSpace API interactions.

2- Test generation of pre-signed URLs for bitstream downloading

Add the following to your .env file and restart ipython shell, with the values shared outside of this PR:

DSPACE_API_BASE="..."
DSPACE_USERNAME="..."
DSPACE_PASSWORD="..."

NOTE: at this time, this application is expecting these env vars to authenticate if API base + username + password is not explicitly passed during DSpaceClient init. Still in flux how this app will receive the credentials. It's possible they will be parsed from another secret, and likely passed to the client init. The client itself will likely remain defaulting to these three env vars for ease of testing.

Get a DSpaceClient instance::

from dfh.dspace import *

dspace_client = get_dspace_client()
# 2026-05-14 14:17:57,542 - Authenticated successfully as *********

Note that authentication is automatic due to auth_on_init defaulting to True in function.

Generate a pre-signed S3 URL for the bitstream shown above:

get_presigned_url_for_bitstream(dspace_client, "b0e3c854-de3e-44ce-94a3-6088615a5eeb")

The link string that is returned is only valid for a single use and for a limited amount of time; both of which are easily satisfied by this harvester as it loops through records and utilizes the URLs.

Includes new or updated dependencies?

YES

Changes expectations for external applications?

NO

What are the relevant tickets?

https://mitlibraries.atlassian.net/browse/USE-557

Code review

Code review best practices are documented here and you are encouraged to have a constructive dialogue with your reviewers about their preferences and expectations.

…ation Why these changes are being introduced: This harvester is designed to retrieve fulltext for DSpace theses in TIMDEX with the goal of ultimately writing it back to the TIMDEX dataset associated with particular records. As such, we can read records from the TIMDEX dataset to use as the driver of what records this harvester will harvest for. Additionally, the original DSpace METS is present in the TIDMEX dataset and provides information about DSpace item bitstreams, including the TEXT bitstream we aim to harvest. How this addresses that need: A new module `dfh.timdex_dataset` is created with a class `TIMDEXThesesRecords` that is designed to find DSpace theses records in the TIMDEX dataset, optionally filtered by `run_id`, and parse information from those record's source DSpace METS about bitstreams. The final result is an iterator of TIMDEX records + bitstream information we can use to harvest the fulltext from DSpace. The objects will be used by this harvester to a) harvest fulltext from DSpace and b) write the fulltext back to the TIMDEX dataset associated with the right record. Side effects of this change: * None really, nothing yet depends on this. Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/USE-557

Why these changes are being introduced: There is one primary interaction with the DSpace API: generating pre-signed S3 URLs of bitstreams to download. To do so, we'll need to authenticate and send GET and/or POST requests. How this addresses that need: Instead of importing a fully featured DSpace 8 / CRIS client, a slimmed down version is created here that does three things well: 1. authenticates 2. supports generic, authenticated GET/POST requests 3. opinionation for the one currently known API request we'll make, generating pre-signed URLs Side effects of this change: * This departs from other applications like DSC and DSS which use a community DSpace 8 / CRIS API library. It would be very straight-forward to pivot to using that in the future for this application if needed, but it seems heavier than what we need at the moment. Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/USE-557

ghukill · 2026-05-13T19:59:11Z

+        self._update_xsrf_token(r)
+        return r
+
+    def get_presigned_url_for_bitstream(self, bitstream_uuid: str) -> str:


Convenience method for a pre-signed URL, leaning on a self.api_get() call.

ghukill · 2026-05-13T20:00:46Z

+    def _init_timdex_dataset(self, dataset_location: str | None) -> TIMDEXDataset:
+        """Init TIMDEXDataset and compose to self.
+
+        If dataset_location is not passed, look to env var TIMDEX_DATASET_LOCATION.


This is the first TIMDEX application that optionally uses a TIMDEX_DATASET_LOCATION env var. The CLI will likely still require it get passed, to align with other TIMDEX apps in the pipeline... but looking to establish this alternate pattern here and now with the hopes perhaps we can move to an env var for most/all apps at some point.

ghukill · 2026-05-13T20:06:05Z

+class TIMDEXThesesRecords:
+    """Class to retrieve TIMDEX dataset records of DSpace Theses.
+
+    Theses are determined by looking for "Thesis" in transformed_record.content_type.
+    """


In the event we broaden our harvesting of fulltext beyond theses, it would not be difficult to add some inheritance here:

TIMDEXRecords

TIMDEXThesesRecords(TIMDEXRecords)

all shared except opionated filtering to theses records only

I mention this only to point out that while we have some theses-opinionation in the business logic here, it will be easy to extend if and when needed. Opted to focus on the immediate need.

Copilot

Pull request overview

Introduces the initial harvesting “selection + lookup” layer by (1) iterating TIMDEX DSpace thesis records, (2) extracting TEXT bitstream metadata from embedded DSpace METS, and (3) providing a lightweight DSpace REST client to generate pre-signed download URLs for known bitstream UUIDs.

Changes:

Add TIMDEXThesesRecords to fetch/filter DSpace thesis records from the TIMDEX dataset and parse TEXT bitstream metadata from METS.
Add DSpaceClient to authenticate against DSpace and request signed URLs for bitstreams.
Add pytest coverage and fixtures for METS parsing edge-cases; adjust lint/typechecker configuration.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
`dfh/timdex_dataset.py`	New TIMDEX dataset helper for filtering thesis records and extracting TEXT bitstream info from METS XML.
`dfh/dspace.py`	New lightweight DSpace REST client for auth + signed URL retrieval.
`tests/test_dataset.py`	New tests covering TIMDEXThesesRecords iteration and METS parsing error cases.
`tests/conftest.py`	Adds fixtures for TIMDEX record JSON + various METS XML inputs.
`tests/fixtures/timdex/dspace-1721.1-139336.json`	TIMDEX thesis record fixture used by dataset tests.
`tests/fixtures/dspace/mets/1721.1-139336.xml`	“Happy path” METS fixture with TEXT fileGrp.
`tests/fixtures/dspace/mets/1721.1-139336-missing-text-filegrp.xml`	METS fixture missing TEXT fileGrp (error case).
`tests/fixtures/dspace/mets/1721.1-139336-text-filegrp-missing-file.xml`	METS fixture with empty TEXT fileGrp (error case).
`tests/fixtures/dspace/mets/1721.1-139336-text-file-missing-flocat.xml`	METS fixture missing FLocat for TEXT file (error case).
`tests/fixtures/dspace/mets/1721.1-139336-text-flocat-missing-href.xml`	METS fixture missing href in FLocat (error case).
`pyproject.toml`	Updates mypy excludes/overrides and ruff configuration (exclusions/ignore list).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ehanson8

Works great and the code looks good, a few suggestions!

ehanson8 · 2026-05-14T15:53:27Z

+    """
+
+
+class TextBitstreamInfo(TypedDict):


Was there a reason to not use FulltextBitstreamInfo? I see you've bounced between text and fulltext in the names, comments, and docstrings

Nice catch - no good reason. I'll try and normalize on "Fulltext".

You know, on second thought, maybe there was rhyme and reason.

We know that we're going after "fulltext" for theses. But technically the bitstream is TEXT.

If you're okay with it @ehanson8, I'd kind of like to keep as-is. Tried to use "fulltext" in places where we're speaking about it conceptually, but often "text" in classes like this and methods that are quite literally interacting with the TEXT bitstream.

That makes sense, a docstring clarifying this could be helpful but not required

Why these changes are being introduced: There is a relatively new DSpace 8 / CRIS client `dspace-rest-client`, that we are using in other projects. We would benefit from normalizing to use that here as well. Even though are needs are lightweight here, it handles the authentication well and provides an identical surface to work from (the "lightweight" client here in code was cribbed heavily from it to begin with). How this addresses that need: * Removes custom `DSpaceClient` class * Replaces with two functions: one to get a `DSpaceClient` instance and a second to generate a pre-signed bitstream download URL using that client Side effects of this change: * Take on dependencies of the 3rd party client `dspace-rest-python`, but also benefit from the community improvements to auth, dependency updates, etc. Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/USE-557

ghukill · 2026-05-15T12:50:46Z

No rush here at all @ehanson8 for re-review, just confirming that we did pivot to using the dspace-rest-python client: 6e35ea1.

ehanson8

Looks great, excellent work!

ghukill added 3 commits May 13, 2026 15:26

Linting and testing updates

52f065a

ghukill commented May 13, 2026

View reviewed changes

ghukill marked this pull request as ready for review May 13, 2026 20:06

ghukill requested a review from a team as a code owner May 13, 2026 20:06

jonavellecuerdo self-assigned this May 13, 2026

ehanson8 self-assigned this May 13, 2026

ghukill requested a review from Copilot May 14, 2026 12:47

Copilot started reviewing on behalf of ghukill May 14, 2026 12:48 View session

Copilot AI reviewed May 14, 2026

View reviewed changes

ghukill added 4 commits May 14, 2026 09:16

Handle METS XML parsing error and continue

cc2e0ca

Fix bistream typo

2b970d2

Raise error on DSpaceClient authentication failure

daa0e44

Explicit add of requests library

fd09ac6

ehanson8 reviewed May 14, 2026

View reviewed changes

ghukill added 2 commits May 14, 2026 13:56

Spelling and naming updates

7b22d32

ghukill requested a review from ehanson8 May 14, 2026 18:32

add ipython as dev dependency

849ceb7

ehanson8 approved these changes May 15, 2026

View reviewed changes

Comment thread dfh/timdex_dataset.py

Comment thread dfh/dspace.py

ghukill mentioned this pull request May 15, 2026

USE 557 - Parallelized input record read and bitstream download #10

Merged

jonavellecuerdo approved these changes May 18, 2026

View reviewed changes

ghukill merged commit d62ee7c into main May 18, 2026
5 of 6 checks passed

Uh oh!

Conversation

ghukill commented May 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose and background context

How can a reviewer manually see the effects of these changes?

1- Test new class TIMDEXThesesRecords

2- Test generation of pre-signed URLs for bitstream downloading

Includes new or updated dependencies?

Changes expectations for external applications?

What are the relevant tickets?

Code review

Uh oh!

ghukill May 13, 2026

Choose a reason for hiding this comment

Uh oh!

ghukill May 13, 2026

Choose a reason for hiding this comment

Uh oh!

ghukill May 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ehanson8 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ehanson8 May 14, 2026

Choose a reason for hiding this comment

Uh oh!

ghukill May 14, 2026

Choose a reason for hiding this comment

Uh oh!

ghukill May 14, 2026

Choose a reason for hiding this comment

Uh oh!

ehanson8 May 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ghukill commented May 15, 2026

Uh oh!

ehanson8 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ghukill commented May 13, 2026 •

edited

Loading

1- Test new class `TIMDEXThesesRecords`

ghukill May 13, 2026 •

edited

Loading

ehanson8 left a comment •

edited

Loading